Since a max(Q) buffer is not applicable in SNO, how about a numeric buffer of 5% (or more?)? That should help alert before things go out of budget.

wouldn't (count(kube_node_info) == 1 and ... mess up the returned value (e.g. it would always return 1)? Given the complexity of the expression, I'd advocate for some unit tests in the first place to assert the current rule.

I'll add some unit tests around this, but I'm not sure why this will always return 1?

foo == 1 and bar will always return 1 (the right-hand side is only used for label matching).

(count by (%(clusterLabel)s) (max by (%(clusterLabel)s, node) (kube_node_info)) == 1 and

sum by (%(clusterLabel)s) (namespace_cpu:kube_pod_container_resource_requests:sum{%(ignoringOverprovisionedWorkloadSelector)s}) - 

sum by (%(clusterLabel)s) (kube_node_status_allocatable{%(kubeStateMetricsSelector)s,resource="cpu"}) > 0)

Apologies if I'm missing something here, but this seems to have boolean expressions on both sides, similar to, say, vector(1) == 1 and vector(2) - vector(1) > 0 (scalar vector on RHS of and instead of an instant vector, so no label matching)?

If this was the aforementioned case, I would've preferred the bool operator to avoid the default filtering behavior, but it seemed to suffice without that.

Had a chat with Simon. We essentially need to swap the LHS and RHS here to make sure the value being returned makes sense for the annotation, as it's currently being used by it (which I missed to notice).

Additionally, we want to hold this a bit and talk a bit more about the usefulness of the alert. As in, in a non-SNO environment, the alert indicates that the user should make accommodations for more resources by, say, adding more nodes, but in SNO, it just ends up being more-or-less of an OOM indicator.

Had a chat with @qJkee, with the conclusion that we want to keep these alerts while modifying them for SNO, as "it still applies to SNO in case if user requests more resources than it has in the cluster. Plus, in that case we can suggest to the users to add worker nodes or increase resources on the given node."

Additionally, Bulat suggested that alerting earlier in case of SNO makes sense, and thus we'd want a ~85% threshold (as usually they are loaded that % and even more because they run on the edge) before firing it.

@simonpasquier I'll go ahead and make these changes, in addition to the ones above, but PLMK if I missed something.

I still feel that reusing the same alert for single node clusters creates confusion. For instance the alert description doesn't fit right in this case:

• By definition, a single node cluster can't tolerate any node failure.
• I should be able to use all the allocatable CPU without interfering with the system components (etcd, kube-api, kubelet, ...) so I'm not sure why we multiply by 0.85. E.g. the reserved CPU value is tuned to the right usage.

I thought about reserving the final 15% as a leeway for backgrounded processes that may belong to userland but not necessarily be initiated by the user, and fall in a "grey" space.

But I agree, this introduces an opinionated constant threshold that doesn't make sense if we look at the metrics and alert definitions. Like you said, "allocatable" should mean that till 100%, and "overcommitment" should quite literally mean exceeding that limit. I'll drop the threshold.

Talking with Balut, I'm inclined to believe that the users will expect the alert to adapt to SNO, as in, if the requests simply exceed the allocatable resources. It may additionally be ambiguous to introduce new alerts for SNO as a 1+1 multi-node system may stop firing AlertX and start firing AlertYSNO when it's reduced to SNO, since the latter is arguably a derivative of the former to some degree (as some users may expect).

(we could of course explore the possibility of adding SNO-exclusive alerts, but I just wanted to put these points out there, I'm on the fence mostly)

To add on, I think we also need to revise how we handle SNO downstream, as in, at the moment we recognize SNO by a SingleReplica topology infrastructure, however, SNO can technically have more than one node (SNO+1 configurations), which would be in-line with having a different set of alerts for SNO while ensuring that upstream expects the same. I'll see if I can find any SNO-dedicated teams or people who can additionally shed some light here.

@rexagod how would you like to proceed with this PR? I've held back from merging as it seems there's open discussion still.

Apologies for the delay here, we are talking this through internally and I'll update as soon as there's a resolution.

FYI the tests have moved into tests/ directory since:

• tests: add more tests #1002